<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Style-Type" content="text/css" />
  <meta name="generator" content="pandoc" />
  <title>nfs_latency</title>
  <style type="text/css">code{white-space: pre;}</style>
  <style type="text/css">
div.sourceCode { overflow-x: auto; }
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
  margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; }
td.sourceCode { padding-left: 5px; }
code > span.kw { color: #007020; font-weight: bold; } /* Keyword */
code > span.dt { color: #902000; } /* DataType */
code > span.dv { color: #40a070; } /* DecVal */
code > span.bn { color: #40a070; } /* BaseN */
code > span.fl { color: #40a070; } /* Float */
code > span.ch { color: #4070a0; } /* Char */
code > span.st { color: #4070a0; } /* String */
code > span.co { color: #60a0b0; font-style: italic; } /* Comment */
code > span.ot { color: #007020; } /* Other */
code > span.al { color: #ff0000; font-weight: bold; } /* Alert */
code > span.fu { color: #06287e; } /* Function */
code > span.er { color: #ff0000; font-weight: bold; } /* Error */
code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
code > span.cn { color: #880000; } /* Constant */
code > span.sc { color: #4070a0; } /* SpecialChar */
code > span.vs { color: #4070a0; } /* VerbatimString */
code > span.ss { color: #bb6688; } /* SpecialString */
code > span.im { } /* Import */
code > span.va { color: #19177c; } /* Variable */
code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code > span.op { color: #666666; } /* Operator */
code > span.bu { } /* BuiltIn */
code > span.ex { } /* Extension */
code > span.pp { color: #bc7a00; } /* Preprocessor */
code > span.at { color: #7d9029; } /* Attribute */
code > span.do { color: #ba2121; font-style: italic; } /* Documentation */
code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
  </style>
  <link rel="stylesheet" href="doc/linux-style.css" type="text/css" />
</head>
<body>
<div id="header">
<h1 class="title">nfs_latency</h1>
</div>
<div id="TOC">
<ul>
<li><a href="#nfs_latency">nfs_latency</a><ul>
<li><a href="#原始代码">原始代码</a></li>
<li><a href="#修改部分">修改部分</a></li>
<li><a href="#最后通过nfs挂载测试">最后通过nfs挂载测试</a></li>
</ul></li>
</ul>
</div>
<h1 id="nfs_latency">nfs_latency</h1>
<p>用bpf获取rpc执行时间戳，计算函数执行时间这个代码是记录每调用一次rpc，就记录调用的rpc时间。代码具体用到了hashmap去存储每个进程号、当前时间戳。用数组去记录进程的pid和对应的耗时时间。</p>
<h2 id="原始代码">原始代码</h2>
<div class="sourceCode"><pre class="sourceCode c"><code class="sourceCode c">nfs.bpf.c
<span class="co">// SPDX-License-Identifier: GPL-2.0</span>
<span class="pp">#include </span><span class="im">&quot;vmlinux.h&quot;</span>
<span class="pp">#include </span><span class="im">&lt;bpf/bpf_helpers.h&gt;</span>
<span class="pp">#include </span><span class="im">&lt;bpf/bpf_tracing.h&gt;</span>

<span class="dt">char</span> LICENSE[] SEC(<span class="st">&quot;license&quot;</span>) = <span class="st">&quot;GPL&quot;</span>;

<span class="kw">struct</span> event {
    u32 pid;       <span class="co">//pid进程</span>
    u64 delta_ns;   <span class="co">//耗时时间</span>
    <span class="dt">char</span> comm[<span class="dv">16</span>];   <span class="co">//进程名</span>
};

<span class="kw">struct</span> {
    __uint(type, BPF_MAP_TYPE_HASH); <span class="co">//指定类型是BPF_MAP_TYPE_HASH ----哈希表</span>
    __uint(max_entries, <span class="dv">4096</span>);          <span class="co">//最大key数量=4096</span>
    __type(key, u64);       
    __type(value, u64);
} start SEC(<span class="st">&quot;.maps&quot;</span>);  <span class="co">//SEC是指定将这个结构体放在BPF的.maps段中，在BPF中，段是一种组织代码和数据的方式</span>

<span class="kw">struct</span> {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);  <span class="co">//数组</span>
} events SEC(<span class="st">&quot;.maps&quot;</span>);

SEC(<span class="st">&quot;kprobe/rpc_call_start&quot;</span>)   <span class="co">//表示这个 BPF 程序会被加载到内核中，并在每次rpc_call_start内核函数被调用时触发执行，也就是rpc_start开始执行时触发</span>
<span class="dt">int</span> BPF_KPROBE(handle_rpc_start)  <span class="co">//BPF_KPROBE是一个宏用于定义一个 kprobe 类型的 BPF 程序</span>
{
    u64 tid = bpf_get_current_pid_tgid();           <span class="co">//获取当前PID和TGID</span>
    u64 ts = bpf_ktime_get_ns();                <span class="co">//获取当前的时间戳</span>
    bpf_map_update_elem(&amp;start, &amp;tid, &amp;ts, BPF_ANY); <span class="co">//向start哈希表里面更新数据，PF_ANY 表示如果键已存在，则更新值；如果键不存在，则插入新键值对</span>
    <span class="cf">return</span> <span class="dv">0</span>;
}

SEC(<span class="st">&quot;kretprobe/rpc_call_done&quot;</span>)
<span class="dt">int</span> BPF_KRETPROBE(handle_rpc_done)
{
    u64 tid = bpf_get_current_pid_tgid();
    u64 *tsp = bpf_map_lookup_elem(&amp;start, &amp;tid);
    <span class="cf">if</span> (!tsp)
        <span class="cf">return</span> <span class="dv">0</span>;

    u64 delta = bpf_ktime_get_ns() - *tsp;   <span class="co">//当前时间戳与hashmap中的时间戳进行计算，得到执行时间。</span>
    bpf_map_delete_elem(&amp;start, &amp;tid);

    <span class="kw">struct</span> event e = {};
    e.pid = tid &gt;&gt; <span class="dv">32</span>;
    e.delta_ns = delta;             
    bpf_get_current_comm(&amp;e.comm, <span class="kw">sizeof</span>(e.comm));  <span class="co">//获取数据存储到event结构体里面</span>

    bpf_perf_event_output(ctx, &amp;events, BPF_F_CURRENT_CPU, &amp;e, <span class="kw">sizeof</span>(e)); <span class="co">//</span>
    <span class="cf">return</span> <span class="dv">0</span>;
}
nfs.c
<span class="pp">#include </span><span class="im">&lt;stdio.h&gt;</span>
<span class="pp">#include </span><span class="im">&lt;unistd.h&gt;</span>
<span class="pp">#include </span><span class="im">&lt;signal.h&gt;</span>
<span class="pp">#include </span><span class="im">&lt;bpf/libbpf.h&gt;</span>
<span class="pp">#include </span><span class="im">&quot;nfs.skel.h&quot;</span>

<span class="dt">static</span> <span class="dt">volatile</span> bool exiting = false;

<span class="co">//用户态</span>
<span class="dt">void</span> handle_event(<span class="dt">void</span> *ctx, <span class="dt">int</span> cpu, <span class="dt">void</span> *data, __u32 size) <span class="co">//BPF 程序输出性能事件时，这个函数会被调用。</span>
{
    <span class="kw">struct</span> event *e = data;
    printf(<span class="st">&quot;PID %d (%s): %.2f ms</span><span class="sc">\n</span><span class="st">&quot;</span>, e-&gt;pid, e-&gt;comm, e-&gt;delta_ns / <span class="fl">1e6</span>);
}

<span class="dt">void</span> handle_lost(<span class="dt">void</span> *ctx, <span class="dt">int</span> cpu, __u64 lost) <span class="co">//当有事件数据丢失时，这个函数会被调用</span>
{
    fprintf(stderr, <span class="st">&quot;Lost %llu events on CPU %d</span><span class="sc">\n</span><span class="st">&quot;</span>, lost, cpu);
}

<span class="dt">void</span> sig_handler(<span class="dt">int</span> sig)
{
    exiting = true;
}

<span class="dt">int</span> main()
{
    <span class="kw">struct</span> nfs_bpf *skel;
    <span class="dt">int</span> err;

    signal(SIGINT, sig_handler);
    signal(SIGTERM, sig_handler); <span class="co">//注册信号处理程序，以便在收到 SIGINT 或 SIGTERM 时调用 sig_handler，</span>

    skel = nfs_bpf__open();
    <span class="cf">if</span> (!skel) {
        fprintf(stderr, <span class="st">&quot;Failed to open skeleton</span><span class="sc">\n</span><span class="st">&quot;</span>);
        <span class="cf">return</span> <span class="dv">1</span>;
    }

    err = nfs_bpf__load(skel);
    <span class="cf">if</span> (err) {
        fprintf(stderr, <span class="st">&quot;Failed to load skeleton</span><span class="sc">\n</span><span class="st">&quot;</span>);
        <span class="cf">return</span> <span class="dv">1</span>;
    }

    err = nfs_bpf__attach(skel);    <span class="co">//将 BPF 程序附加到内核中的 kprobe 和 kretprobe 点</span>
    <span class="cf">if</span> (err) {
        fprintf(stderr, <span class="st">&quot;Failed to attach</span><span class="sc">\n</span><span class="st">&quot;</span>);
        <span class="cf">return</span> <span class="dv">1</span>;
    }

    printf(<span class="st">&quot;Tracing NFS client RPC latency... Ctrl+C to exit</span><span class="sc">\n</span><span class="st">&quot;</span>);

    <span class="kw">struct</span> perf_buffer *pb = NULL;
    pb = perf_buffer__new(bpf_map__fd(skel-&gt;maps.events), <span class="dv">8</span>,
                          handle_event, handle_lost, NULL, NULL);  <span class="co">//创建缓冲区来接收程序输出的事件数据</span>
    <span class="cf">if</span> (!pb) {
        fprintf(stderr, <span class="st">&quot;Failed to open perf buffer</span><span class="sc">\n</span><span class="st">&quot;</span>);
        <span class="cf">return</span> <span class="dv">1</span>;
    }

    <span class="cf">while</span> (!exiting)
        perf_buffer__poll(pb, <span class="dv">100</span>);  <span class="co">//轮询缓冲区，等待事件数据的到来， poll轮询可以快速响应。</span>

    perf_buffer__free(pb);
    nfs_bpf__destroy(skel);
    <span class="cf">return</span> <span class="dv">0</span>;
}</code></pre></div>
<h2 id="修改部分">修改部分</h2>
<h3 id="error-当前目录下没有-vmlinux.h-文件这个文件在-nfs.bpf.c-中被引用">（1）error: 当前目录下没有 vmlinux.h 文件，这个文件在 nfs.bpf.c 中被引用</h3>
<p><code>sudo bpftool btf dump file /sys/kernel/btf/vmlinux format c &gt; vmlinux.h</code> 这条命令它从运行中的 Linux 内核中提取类型信息，并将其转换为 C 语言头文件格式</p>
<h3 id="error-struct_event没有在nfs.c定义">（2）error: struct_event没有在nfs.c定义</h3>
<div class="sourceCode"><pre class="sourceCode c"><code class="sourceCode c"><span class="kw">struct</span> event {
    __u32 pid;       <span class="co">// 进程 ID</span>
    __u64 delta_ns;  <span class="co">// 耗时时间</span>
    <span class="dt">char</span> comm[<span class="dv">16</span>]; <span class="co">// 进程名</span>
};</code></pre></div>
<p>加入到nfs.c中。</p>
<h3 id="perf_buffer__new-函数调用的参数数量不正确我们使用的参数是-6-个但函数定义只接受-3-个参数-那么我们来看一下函数正确的参数">（3）perf_buffer__new() 函数调用的参数数量不正确，我们使用的参数是 6 个，但函数定义只接受 3 个参数 那么我们来看一下函数正确的参数：</h3>
<div class="sourceCode"><pre class="sourceCode c"><code class="sourceCode c">perf_buffer__new(<span class="dt">int</span> map_fd, <span class="dt">size_t</span> page_cnt,
                 <span class="dt">const</span> <span class="kw">struct</span> perf_buffer_opts *opts);</code></pre></div>
<p>来看一下perf_buffer_opts结构体</p>
<div class="sourceCode"><pre class="sourceCode c"><code class="sourceCode c"><span class="kw">struct</span> perf_buffer_opts {
        <span class="co">/* if specified, sample_cb is called for each sample */</span>
        perf_buffer_sample_fn sample_cb;
        <span class="co">/* if specified, lost_cb is called for each batch of lost samples */</span>
        perf_buffer_lost_fn lost_cb;
        <span class="co">/* ctx is provided to sample_cb and lost_cb */</span>
        <span class="dt">void</span> *ctx;
};</code></pre></div>
<p>其实就可以看出，调用的API函数已经更新了，只有三个参数，新版本的API，是把事件回调、丢失事件回调、用户上下文放到结构体里面，然后进行结构体指针传参。</p>
<h3 id="内核中有-rpc_call_start-函数但没有-rpc_call_done-函数导致程序.nfs无法加载">（4）内核中有 rpc_call_start 函数，但没有 rpc_call_done 函数。导致程序.nfs无法加载</h3>
<p>同样可能是内核版本不同，将 rpc_call_done 改为 rpc_task_call_done</p>
<h3 id="map-events-创建失败的问题">（5）map 'events' 创建失败的问题</h3>
<p>map 'events': failed to create: Invalid argument 从这句提示可以看出 events的结构体定义不对，map定义，添加更多特定参数以满足内核要求：</p>
<div class="sourceCode"><pre class="sourceCode c"><code class="sourceCode c"><span class="kw">struct</span> {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);  <span class="co">//数组</span>
    __uint(key_size, <span class="kw">sizeof</span>(<span class="dt">int</span>));
    __uint(value_size, <span class="kw">sizeof</span>(<span class="dt">int</span>));
    __uint(max_entries, <span class="dv">128</span>);
} events SEC(<span class="st">&quot;.maps&quot;</span>);</code></pre></div>
<h3 id="无法找到-rpc_task_call_done-函数进行-kretprobe">（6）无法找到 rpc_task_call_done 函数进行 kretprobe</h3>
<p><code>sudo ls -la /sys/kernel/debug/tracing/events/sunrpc/</code> 输入指令后发现了 /sys/kernel/debug/tracing/events/sunrpc/rpc_task_call_done，这表明我们可以使用 tracepoint 而不是 kprobe,因此修改。</p>
<h3 id="由此可以看出大部分都是版本问题">（7）由此可以看出大部分都是版本问题</h3>
<ol style="list-style-type: decimal">
<li>我的libbpf版本是0.5.0， ubuntu的内核版本是6.8.0-59。</li>
<li>perf_buffer API 变化，使用结构体传参。</li>
<li>kprobe 到 tracepoint 的迁移、旧版本: 使用 kprobe/kretprobe 跟踪特定函数，新版本: 使用 tracepoint 跟踪标准化的事件点</li>
<li>Map 定义的完整性要求,旧版本: 最小化定义可能足够,新版本: 需要更完整的定义，包括 key_size, value_size, max_entries</li>
</ol>
<h2 id="最后通过nfs挂载测试">最后通过nfs挂载测试</h2>
<h3 id="本地挂载-nfs-共享">本地挂载 NFS 共享</h3>
<pre><code>             
# 挂载本地共享 (使用NFSv4协议)
sudo mount -t nfs -o vers=4.1 localhost:/s_test /mnt                
# 验证挂载
df -hT | grep nfs4
# 应输出类似：
# localhost:/s_test 526802432 43963904 456005120    9% /mnt            
# 写操作
cd /mnt
touch file
dd if=/dev/urandom of=/mnt/file bs=1M count=10 status=progress</code></pre>
<h3 id="运行脚本文件">运行脚本文件</h3>
<pre><code>sudo make
sudo ./nfs
#先运行./nfs 自去进行写操作
#输出结果如下：
Tracing NFS client RPC latency... Ctrl+C to exit
PID 14807 (ls): 49.01 ms
</code></pre>
</body>
</html>
